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The Intelligent Robotics Group (IRG) 
at NASA Ames Research Center 
develops robotic rovers as research 
platforms for autonomous navigation, 
human robot interaction, and robot 
software. 

Often we send a small team of 
scientists and robot engineers into the 
field with some rovers, and science and 
engineering backroom teams remotely 
operate the rovers. 



As such, it is necessary to visualize 
where the rover is, what algorithms it 
is executing, and what scientific data it 
is gathering. 
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KIO Red at Haughton Crater, 
Devon Island CA 
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Concept of Operations 
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VERVE is an integrated 3D view built on top of Ardor3D. 


We have created a number of tools to assist with remote 
situational awareness. Some of these are web tools, and quite a 
few are Eclipse RCP applications, based around VERVE. 
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We develop and work with many different robots for different needs, and 
attach various scientific instruments to them. We also collaborate with 
different NASA centers to support remote operation of their robots. 


IRG’s K-series rovers have four- 
wheel drive, all wheel steering 
and a passive averaging 
suspension. Navigational sensors 
include a GPS system, digital 
compass, stereo hazard cameras, 
and an inertial measurement unit. 
K-series rovers run Rover 
Software, which supports 
autonomous navigation and 
obstacle avoidance. 



Krex at Basalt Hill, CA 
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The VERVE 3D View is an Eclipse view 
with contents rendered by the 
Ardor3D libraries which we have 
wrapped in a plug-in. 

Ardor3D provides user interface 
support within the view, eg the 
compass rose and overlay text. 

Ardor3D provides hooks for mouse 
control and keyboard input, along 
with typical controls for a 3D graphics 
library (ie cameras, scene, lighting, 
etc). It supports standard 3D file 
formats. 
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A simple model of K I ORed; we 
keep the models small and efficient 
and only articulate what will give us 
useful information. 



The Ardor3D scene graph is comprised of 

Spatials to define geometry and rendering settings 
Nodes spatials with parents and children 

For each type and instance of a robot, we create a RobotNode (extends Node) 
which contains nodes that represent its model (3D object) and child nodes for 
representation of scientific data. 

We have a reference between our conceptual representation of a robot and each of 
its parts, and theArdor3D nodes which represent each concept. 
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public class RobotNode extends Node { 

final AbstractRobot m_robot; 

RobotModelNode m_model; 

Node m_concepts; 

Node m_sensors; 

Node m_base; 

public RobotNode (AbstractRobot robot , Node model) { 
super ( robot . getName ( ) ) / 

m_robot = robot ; 
setRobotModel (model ) ; 

} 
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We have created a simple interface to connect the 
incoming telemetry to the robot parts that are 
rendered: 


/ * * 

* interface for scenegraph -> telemetry glue for robot visualization parts 
*/ 

public interface IRobotPart { 

public String getPartName ( ) ; 
public AbstractRobot getRobot(); 

public void connectTelemetry ( ) throws TelemetryException; 
public void disconnectTelemetry ( ) throws TelemetryException; 

public void attachToNodesIn(Node model) throws TelemetryException , 
IllegalStateException; 

public void handleFrameUpdate ( long currentTime) ; 
public void reset(); 

public boolean isVisible(); 

public void setvisible (boolean state); 

} 
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Desert RATS featured several robots, ^/ehicles, and 
astronauts with sensors 
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A model of a space suit with instrument backpack and the 
Centaur 2 rover, which we worked with at Desert RATS 
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We have a standard Eclipse view 
which includes a tree populated with 
the contents of the scene graph. This 
tree can be extremely deep. 

Since elements are dynamically added 
to and removed from the 3D scene, 
the tree must be populated with 
WeakReferences to support garbage 
collection. 



When various events happen, the tree 
refreshes asynchronously. 


Checkboxes show and hide 3D 
models in theArdor3D view by 
setting their render state. 
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if (show){ 

spatial . setCullHint ( CullHint . Inherit ) / 

} else { 

spatial . setCullHint ( CullHint . Always ) ; 

} 
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We use Java reflection and SWT 
databinding to automatically 
generate SWT Ul components 
based on the Java objects defined 
in the robot class; manipulating 
these widgets affects how the 
components are rendered in the 
Ardor3D scene. 

This allows our robot engineers 
and scientists to customize the 
visualization based on what they 
are looking for. 
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Typically we work with rovers to explore terrain that has not been 
well mapped. We may have satellite imagery and information about 
the terrain. This data is at a low resolution and may not be 
accurate. As the rovers traverse the area, they build more accurate 
maps. 

We use LIDAR, a remote sensing technology that measures 
distance with a laser. We use LIDAR both for a line scan and to 
return a point cloud. We also do stereo reconstruction with our 
two hazard cameras. 

When we get this information back to our Eclipse RCP application 
from the rover, we can “ground truth” or adjust the terrain to the 
discovered data. 
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Black Point Lava Flow as a GeoTIFF with levels of detail, rendered with haze 
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You can see the line scan and point cloud data that Centaur 2 has 
captured as it drives across a parking lot. Red arrows show the path 

it has taken. 
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We use DDS, the Data Distribution Service for Real-Time Systems as our 
middleware. By having this standard, we can create IDL (Interface Definition 
Language) files that describe the data which is being serialized between 
publishers and subscribers. We define data formats for commands and for 
telemetry, and various consumers (robots, software applications) can subscribe 
to the published topics. 

We configure DDS to handle poor connectivity and dropouts, since we are 
simulating space missions which will have these issues. 

Adhering to the DDS standard makes it more straightforward to integrate with 
various robots and scientific instruments. 
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We used DDS to have astronauts on the International Space Station control our KIO 

NASA NASA VERVE: Interactive 3D Visualization within Eclipse Rover at NASA AmeS /g 



To facilitate collaboration between NASA centers, we came up 
with a standard called RAPID, built on top of DDS. This includes a 
set of IDL files and Java libraries for common commands and 
telemetry. 


http://rapid.nasa.gov/ 
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For our RAPID VERVE implementation, we have a base 
class to connect robot parts to RAPID telemetry. 


public abstract class RapidRobotPart extends AbstractRobotPart implements 
IRapidMessageListener 

{ 

^Override 

public void connectTelemetry ( ) throws TelemetryException { 
if (isTelemetryEnabled( ) ) { 

try {for (Mess ageType msgType : rapidMessageTypes ( ) ) { 

RapidMessageCollector . instance ( ) . addRapidMessageListener ( getParticipantld ( ) , 

getRapidRobot ( ) . getAgent ( ) , msgType, this ); 


This way, when the telemetry comes in, the relevant 
data can be cached and when it is time to rerender, 
handleFrameUpdate is called. 
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We have data coming in too fast to render, so we throttle that back: 


timer = new Timer(); //an Ardor3D timer which uses System. nanoTime 

frameHandler = new FrameHandler (timer ) ; 

frameHandler .addUpdater (new LogicalLayerUpdater ( ) ) ; IT I Li * ^ * 


//we expose camera controls in the Eclipse UI 

frameHandler .addUpdater (new CameraControlUpdater ( ) ) ; I — 

renderUpdateThread = new RenderUpdateThread ( applicationPlugin, 

frameHandler, timer); 

renderUpdateThread. start ( ) ; 


Our very simple render update thread runs as follows: 

Once the system and display are ready (until the display is disposed) 

Check the time that the last change occurred; 

if the elapsed time is enough, asynchronously run an update 


frameHandler . updateFrame ( ) ; 
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Sample handleFrameUpdate for joint data 


public void handleFrameUpdate ( long currentTime) { 
if (isDirty ( ) ) { 
try { 

float [] ja = m_angleData; 

Jointlnfo joint; 

for(int i = 0; i < m_ joints . length; i++) { 
if(i < m_angleDat a . length) { 

joint = m_ joints [ i ] ; 

if( joint . spatial != null && lFloat.isNaN(ja[i]) ) { 

m_rot . f romAngleNormalAxis ( ja[ i ]+ joint . offset, 

joint . rotAxis ) ; 

joint . spatial . setRotation ( m_rot ) ; 

} 

} 

} 

} 

catch (Throwable t) { 

logger. log( t ) ; 

} 

setDirty ( false) ; 

} 

} 
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Scientific Visualization 


Here we are rendering data coming back from a Ground Penetrating 
Radar (GPR) system, so scientists can analyze information about the 
substrate. They can customize the rendering of the GPR telemetry. 


Arrows indicate 
previous position, 
and the rings 
indicate the sun 
tracker and the 
magnetic compass. 
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We also work with the SPHERES on 
the International Space Station (ISS); 
we run DDS on an Android 
Smartphone which we connect to the 
SPHERES, and view and control it from 
our Eclipse RCP Application. 

In our experiment in 20 1 3, we 
simulated an internal inspection of an 
ISS module . 

We have commanded it from the 
ground and astronauts have 
commanded it from the ISS. 
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Screenshot of the SPHERES Smartphone Workbench: Manual Control 
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Deployment of a Telescope 


The Surface Telerobotics experiment in 2013 examined how 
astronauts in the ISS can remotely operate a surface robot (KIO 
Rover) across short time delays. We simulated an astronaut 
teleoperating a rover on the lunar farside to deploy a low radio 
frequency telescope. 



NASA 


NASA VERVE: 


Interactive 3D Visualization within Eclipse 


26 



Rover running a Route Plan; Panorama coming in 
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Manually moving the rover forward and inspecting the film 
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During our Surface Telerobotics experiment, we worked with a 3 rd 
party company (TRACLabs) to subscribe to DDS messages providing 
informational alerts about the state of our rover. We displayed these 
in a growl-like overlay in the VERVE 3DView. 
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Surface Telerobotics Experiment in action: 
Luca Parmitano on the ISS operating KIO at Ames 
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Intelligent Robotics Group 
at NASA Ames Research Center 


• K 1 0 Rover among others 

• SPHERES 

• xGDS Ground Data Systems 

• VERVE 3D within Eclipse 

• Contributed the moon to Google Earth 

• Mars-o-vision (mars.planetary.org) 

• GigaPan robotic camera 

• GeoCam disaster response 



• Ames Stereo Pipeline http://irg.arc.nasa.gov 

• Vision Workbench 

• Tensegrity research http://sourceforge.net/projects/irg-verve/ 


... and more! 
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